🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Enforce the “immutable original” invariant.

Keeping the attachments mount out of the sandbox prevents agent access, but the proposal still reuses a write-by-destination-path upload flow. Define server-side write-once semantics—or immutable object IDs/versioning—and reject overwrite/delete operations. Otherwise the client or another API path can mutate the source of truth, breaking download and findability guarantees.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Define re-materialization after working-copy deletion.

The design allows the agent to edit or delete the working copy, but later turns “skip” materialization unconditionally. Reusing the reference after deletion can therefore leave the agent without the file. Specify an existence check and rehydration policy, while ensuring existing edits are not overwritten unexpectedly.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Specify the capability contract across every layer.

The supplied runner code currently defines images and fileAttachments; this proposal introduces images, audio, and documents. Explicitly update the runner flags/types, Agenta protocol/DTOs, API response, and FE metadata, including the embeddedContext → documents mapping. Otherwise the composer and runner cannot enforce the same gate.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject unsupported modalities instead of dropping them.

“Drop with an explicit warning” still lets the run continue without content the user attached, violating the stated “never silently dropped” and honest-capability goals. Fail the turn with a user-visible structured capability error, rather than invoking the model with a partial prompt.

what are the contentblock types in the vercel adapter?

are there limits we should know about

🤖 The AI agent says:

The Vercel adapter (sdks/python/agenta/sdk/agents/adapters/vercel/messages.py) produces five content block types: text, image, resource, tool_call, and tool_result. For an incoming file part the rule is: a media type starting with image/ becomes an image block, and everything else (PDF, audio, any document) collapses into a resource block. There is no audio block today.

Changed in the PR: research.md section 10 documents the adapter's full vocabulary, the file rule, and the missing audio type. Adding a first-class audio block is part of the plan.

🤖 The AI agent says:

Yes, two kinds of limits. Our own front-end limits (5 files, 5 MB each, image/PDF/text/JSON) are placeholders; the code comment on them says they were meant to be replaced with limits derived from the selected model. The real limits come from the providers: Claude caps a request at about 32 MB, image size limits depend on the deployment, and PDF support goes up to 600 pages (100 pages on 200k-context models). Gemini has its own per-request limits.

Changed in the PR: research.md section 3 lists the provider limits with sources, section 10 covers our placeholders and where they came from, and the plan derives the front-end limits from the selected model instead of the placeholders.

pleaes let us not limit ourselves to image in this research. let us be as general as we can. images, zips, cad files, excel files, folders, videos.. different sizes etc..

🤖 The AI agent says:

Done. research.md section 3 now covers images, audio, video, PDFs, office documents (Excel, Word), plain text and code, CSV and JSON, zip archives, CAD and other binary formats, folders, and very large files.

The organizing idea that came out of it: a model natively perceives only a small set of kinds (images, audio for some models, documents for some, video only for Gemini). Everything else is a file the agent opens with tools, sometimes after converting it (a spreadsheet through a library, a zip by extracting). The design therefore puts every attached file on the agent's disk for tool use, and additionally delivers the native kinds to the model inline when the model supports them.

why do we have two types only / imges and fileattachements

🤖 The AI agent says:

Because the flags were built ahead of the feature that would use them. The whole capability set, including images and file_attachments, landed with the original agent-runtime protocol commit (b9e62f99aa). The docstring even says adapters should "skip image blocks without images", but that skip was never built, because nothing sends image blocks yet. Only the tool flags (mcp_tools, tool_calls) ever got a real gate.

Changed in the PR: research.md section 9 has this history, and the design replaces the two-flag set with a layered capability model (protocol, adapter, model) described in design.md.

again why capabilities are abut images only

🤖 The AI agent says:

Same root cause as the previous comment: the runner probes the harness, maps only images and fileAttachments out of the ACP capability set (image, audio, embeddedContext), and then never reads them anywhere.

Changed in the PR: the design now requires the runner to map all three ACP capabilities, and treats the effective capability for a modality as the intersection of three layers: what the protocol can carry, what the harness adapter actually delivers natively, and what the selected model perceives. The layered contract table is in design.md.

what is a ressource / ressource_link
why ours have no audio.

please update this whole thing and explain the different modalities and how they are usually handled. which ones are special which one are just files

special as in native and require special path

🤖 The AI agent says:

Definitions first. An embedded resource carries the file's bytes inline in the message (as text or base64), so the harness can hand the content to the model. A resource_link is only a pointer (a URI and a name); the agent may fetch it with a tool, but the model is never guaranteed to see it. Our content block simply has no audio variant today (audio collapses into resource); the plan adds one, because ACP represents audio as its own content type and there is no tool-read fallback for it.

Changed in the PR: research.md section 3 is now a full taxonomy of modalities: the native kinds, where the model's own encoder must receive the bytes through message content (images, audio, documents for some models, video for Gemini), versus everything else, which is just a file the agent opens with tools. Section 4 explains each ACP content type and what our two pinned harness adapters actually do with each one.

🤖 The AI agent says:

Exactly. "Special" means native: the model's own encoder must receive the bytes through the message content, and no tool that reads a file from disk can substitute for that path. research.md section 3 uses this as the dividing line: native kinds need the dedicated delivery path, everything else is a normal file for the agent's tools.

please add a research file research.md using the plan-feature skill and make sure to explain exactly the capabilities / endpoints we have in mounts. who has the right to use them. what information are required to use them (arguments). this will help us understand the flows in which they would be used. for instance how they would be used in the drawer (to updload / edit / remove / move files within the mount) or in the message and who has the responsibility to do things. same for the previous context we need research about the different modalities and which ones are special. research about how acp works with these modalities. research about how zed deal with these in acp, about opencode how it deals with these (especially in the ui) and any opensource solution that might have this. to understand best practices flows and responsabilities. also we need to udnerstand capabilities of models and what makes things special

🤖 The AI agent says:

research.md now exists and covers all of this.

Mounts: section 2 has a table of every endpoint (route, method, required permission, arguments, what it does), covering create, query, sign, upload, write, download, list, delete, folder creation, and export. Two facts worth calling out: there is no move or rename endpoint (a move is a write to the new path plus a delete of the old one), and there are two separate byte paths (through the API with permission checks, or directly against S3 with signed credentials, which today are always read-write). Section 2 also maps the endpoints to the three flows and who owns each: the drawer (front end calls upload, write, delete, list; the API enforces permissions), the message attachment (front end uploads once, API stores and validates, runner resolves), and materialization (runner writes the working copy).

Modalities: section 3 (which are native, which are just files, provider limits). ACP: section 4 (each content type, what is required, and what our two pinned adapters actually do with each block). Zed and opencode: section 6 (how each collects files, how they reach the model, who enforces limits). Model capabilities: section 3.

you are conflating context, research, proposal and design. please reorganize the research in a way that is readable by a human being

🤖 The AI agent says:

Restructured. The folder now follows the plan-feature layout: context.md is the plain story only (what happens today, what we want), research.md holds all findings organized by question, design.md is the design written as options and decisions with the diagrams explained in prose, plan.md is the staged work, scope.md is in and out of scope, decisions.md is the compact decision log plus open questions, and status.md tracks state. proposal.md is deleted; its content was split into design.md and plan.md. README.md opens with a glossary and a reading order.

this is a limitation pre-session work. after sessions we will send only the last message. even in case of catastrophic failure the responsability of sending all messages will go to the runner

basically in case the session cannot be continued the runner should read from records and then rerun them (let's create an issue for this in case we did not already)

🤖 The AI agent says:

Agreed, and it is now framed that way. research.md section 7 describes resending the full history as the pre-sessions behavior, and states the direction: the front end sends only the last message, and the runner owns rebuilding context from records when a session cannot be resumed warm.

One verified fact that matters for the plan: records today are text-only (the record schema allows only {type: "message", text}), so carrying attachment references in records requires extending the record schema. The plan makes that extension part of the first release, so the records are ready for the runner-rebuilds direction.

🤖 The AI agent says:

There was no existing issue, so I created one: #5443, "Runner should rebuild session context from records when a session cannot be resumed warm". It links #5384 (the warm case: hold client tool calls open instead of replaying) and this PR (records will carry attachment references, which a faithful rebuild needs). research.md section 7, scope.md, and status.md link to it.

me no understand

please always write using context (update for the whole document)

🤖 The AI agent says:

Rewritten in plain words. What it means: saved chat messages currently keep the file's bytes inline, encoded as text. The browser gives a site about 5 MB of local storage. One attachment can exceed that, and when the save routine runs out of room, it deletes other sessions' saved chats to make space. So attaching one file in one conversation can silently erase the saved history of other conversations.

Changed in the PR: context.md and research.md section 1 explain it this way, and the fix falls out of the design: once messages store a small reference instead of the bytes, the storage pressure is gone.

🤖 The AI agent says:

Applied across the folder. README.md now carries a glossary of the domain terms the files lean on (runner, harness, adapter, ACP, mount, composer, materialize, records, attachment resource, warm and cold sessions, and so on), the main flows introduce each entity together with which system owns it and where it runs, and the known coinages were removed or replaced with plain words. If you still hit a term that is used before it is explained, flag it and I will fix that spot.

this should be taken into accoun tin the plan

🤖 The AI agent says:

Done. plan.md folds the tracing change into the first release: once messages and records carry a reference instead of bytes, the Python agent span (which keeps messages on purpose and has no length cap) holds a small reference rather than a base64 blob. The release verifies that outcome instead of adding a truncation cap, since the cause is removed at the source.

I would add also large copy pasta. At least as a follow up. If the user copy pastes a large document, we should not send it though the prompt, it should instead be sent as a file (question where would it be saved maybe answered elsewhere) and then referenced in the prompt instead.

when we upload something we should show the reference in the prompt in a nice way (out of scope probably but follow up)

🤖 The AI agent says:

Added as a follow-up in scope.md: when a person pastes a very large block of text, it should be saved as a file and referenced in the prompt, the same path an attachment takes. On your "where would it be saved" question: the attachments mount this project builds is the leading option, since the reference mechanism would be identical, but the follow-up still needs its own storage decision when it is picked up.

🤖 The AI agent says:

Added as a follow-up in scope.md: a clean reference chip in the composer (filename and kind) when a file is attached, as a presentation improvement on top of the core flow.

why are we saying that they should reach the model as standard acp blocks. what are the advantages here instead of dealing with it ourselves (that is uploading to the mount then referencing).

Please add research to explain how does ACP uses these blocks. what happens under the hood when communicating with the harness. which harnesses actually work/use these acp blocks. and how difefrent is it from the other approcah i mentioned

🤖 The AI agent says:

The research now answers this with the actual adapter code, and the answer is: we need both paths, because they do different jobs.

Under the hood: an ACP image or audio block carries required base64; the harness adapter converts it into the model API's native content. We verified the two adapters we actually pin: the Claude adapter (@agentclientprotocol/claude-agent-acp 0.58.1) converts ACP images into native Claude image blocks, and Pi (pi-acp 0.0.29) passes images through as image inputs. So inline ACP blocks are the reliable way to make the model see an image.

The alternative you describe (upload to the mount, mention the path) gives the agent tool access but not reliable perception: ACP has no "model reads a URL" content type, and Claude Code's Read tool does not dependably return an image as vision input. For audio there is no tool fallback at all.

So the design does both: upload to the mount and reference (for storage, findability, and the agent's working copy) plus inline delivery at prompt time (for perception, rebuilt by the runner from the stored original). research.md sections 4 and 5; design.md decision D1.

not sure about never lost. if the file is uploaded this means it can be deleted

🤖 The AI agent says:

You are right; the wording overclaimed. The guarantee is now stated precisely: the agent cannot change or delete the original, because the attachments mount is never exposed in the sandbox and originals only enter through a create-only upload route (the API refuses overwrite and delete for originals, and no signed storage credentials are issued for that mount). It is not immortality: Stage 1 defines cleanup for uploads that were never sent, and what happens on session archive or delete is an open retention question in decisions.md. design.md decision D7.

what does it mean to have unsopported modalities? an agent using an llm that does not support images can still use tools that use images. no? please explain in research and address this comment

🤖 The AI agent says:

Correct, and this comment changed the design. Perception and tool use are now treated as two separate things. "Unsupported modality" only ever means the model cannot natively perceive that kind. The file still lands on the agent's disk, so tools work regardless of the model.

The gating now works accordingly: attaching a file always puts it in the agent's workspace; native delivery to the model happens when the capability intersection (protocol, adapter, model) allows it; when it does not, the composer says so up front and the turn proceeds with the file as a workspace-only attachment. The runner fails the turn only when it is explicitly asked to emit a native block the harness cannot accept. research.md section 3; design.md decisions D5, D6, and the open decision D11 (what attaching promises in the first release).

please be very precise here. what do you mean by hot path, what do you mean by history. to my knowledge we dont have an entity called history. and why do we care about the wire between the service and the runner not having base64, what is the value there?

🤖 The AI agent says:

Rewritten without those words. Precisely: the wire is the request from the front end through the API to the runner. There is no entity called "history"; the two real things are the front end's saved message store (browser local storage) and the durable per-session records kept by the API. The docs now name those.

Why carrying base64 there hurts: the front end currently resends the whole conversation on every turn, so the same image bytes travel again on every later turn (the cost grows with the square of the conversation length); the saved bytes overflow browser storage and evict other sessions' chats; and the bytes land verbatim in traces. A small reference removes all three. That is the value, and it is separate from the model boundary, which still needs inline bytes at prompt time. research.md section 1.

again not sure i understand the acp content model and what does it actually mean for us

🤖 The AI agent says:

research.md section 4 now lays it out concretely: the five ACP content types, their exact fields, and which fields are required. What it means for us is one fact: for the model to perceive a file, the bytes must be inline in the turn at the moment the runner calls the harness. Storage never removes that requirement; it only removes bytes from the wire and the saved state. And one caveat we verified in the pinned adapters: what the model actually receives depends on the adapter, not just the protocol (images arrive natively; documents and audio do not today).

can you explain in the research what are these assistant emitted files. how the flow look like there. where is this even used. i would expect an agent to just write the files to the mount and reference them in its answer (by the way how does the frontend know when to show a link to a file [in agenta but also in zed])

🤖 The AI agent says:

research.md section 8 covers this. The runner protocol declares a file event ({type: "file", url, mediaType}), but no runner code emits it. Your expectation matches how Zed and opencode actually behave: the agent writes files into its working directory and mentions them in its answer, and the client shows the file because it can see the working directory, not because of a protocol event. In Agenta the drawer already lists cwd, so an agent-produced file is already findable there; the missing piece is an inline "here is the file" chip in the answer, which is separate work and listed in scope.md as out of scope here.

please explain what are these dont talk about anything without defining it and explaining th econtext. what the hell is a content-addressed dedup or transcoding and why does it matter

🤖 The AI agent says:

Defined now, each in one sentence in scope.md. Deduplication by content: store one copy when the same file is shared twice (a storage saving, not a correctness need). Transcoding: converting a file's format before delivery, for example turning a video into frames. Thumbnails: server-made preview images for large files. All three are deferred optimizations that do not block the core flow, which is why they are out of scope.

you need to precise here. what do you mean by attachement reference. you are providing a json without any context to where it is used. is it in vercel? which part of the system. please explain the flow from a to z when you explain something and explain the boundaries of the system and the responsabilities of each tihng. explain high level but also mentioning the entities and primitives we have, this is in no a way an explanation a staff engineer would use.

🤖 The AI agent says:

Rewritten, and the shape itself changed during review. The reference is no longer a raw {mount_id, path, ...}: the API now issues an opaque attachment resource on upload (attachment_id plus server-verified filename, media type, and size). The front end sends the attachment_id in the message parts in place of today's base64 file part; it rides the /run wire as a content block; it is what records persist; and the runner resolves it through the API, which checks the attachment belongs to the session. Raw storage coordinates would have let a client forge references to other sessions' files and made client-supplied metadata authoritative.

design.md now walks the flow end to end with every system named at each step (front end, API, object store, runner, sandbox, harness, model) and what each is responsible for; research.md section 2 has the responsibility split per flow; decision D10 records the shape choice.

what does the different right even mean

🤖 The AI agent says:

Removed the phrase. The concrete difference: the original cannot be changed by the agent (it is outside the sandbox and only enters through a create-only API route), and the working copy can be changed freely (it sits in the working directory, which the agent fully controls). design.md decision D4 states it this way.

why did we decide to have it immutable and agent unreachable etc.. what is the reason / constraint behind this design?

🤖 The AI agent says:

The reason is the findability goal: a person must always be able to return to exactly what they shared. The agent's working directory is last-writer-wins, and its own README says concurrent runs overwrite freely. An agent asked to "clean up the workspace" can delete or overwrite anything there. Keeping the original where the agent cannot reach it is the only way the guarantee holds no matter what the agent does. design.md decision D3 spells this out with the concrete failure scenario.